The stimulus-driven and representation-driven cross-modal attentional spreading are both modulated by audiovisual temporal synchrony.

Multisensory integration and attention can interact in a way that attention to the visual constituent of a multisensory object results in an attentional spreading to its ignored auditory constituent, which can be either stimulus-driven or representation-driven depending on whether the object's visual constituent receives extra representation-based selective attention. Previous research using simple unrelated audiovisual combinations has shown that the stimulus-driven attentional spreading is contingent on audiovisual temporal simultaneity. However, little is known about whether this temporal constraint applies also to the representation-driven attentional spreading, and whether it holds for the stimulus-driven process elicited by real-life multisensory objects. The current event-related potential study investigated these questions by systematically manipulating the visual-to-auditory stimulus onset asynchrony (SOA: 0/100/300 ms) in an object-selective visual recognition task wherein the representation-driven and stimulus-driven spreading processes, measured as two distinct auditory negative difference (Nd) components, could be isolated independently. Our results showed that both the representation-driven and stimulus-driven Nds decreased as the SOA increased. Interestingly, the representation-driven Nd was completely absent, whereas the stimulus-driven Nd was still robust, when the auditory constituents were delayed by 300 ms. These findings not only indicate that the role of audiovisual simultaneity in the representation-driven attentional spreading has been underestimated, but also suggest that learned associations between the unisensory constituents of real-life objects render the stimulus-driven attentional spreading more tolerant of audiovisual asynchrony.

The dissociation of semantically congruent and incongruent cross-modal effects on the visual attentional blink.

Introduction: Recent studies have found that the sound-induced alleviation of visual attentional blink, a well-known phenomenon exemplifying the beneficial influence of multisensory integration on time-based attention, was larger when that sound was semantically congruent relative to incongruent with the second visual target (T2). Although such an audiovisual congruency effect has been attributed mainly to the semantic conflict carried by the incongruent sound restraining that sound from facilitating T2 processing, it is still unclear whether the integrated semantic information carried by the congruent sound benefits T2 processing. Methods: To dissociate the congruence-induced benefit and incongruence-induced reduction in the alleviation of visual attentional blink at the behavioral and neural levels, the present study combined behavioral measures and event-related potential (ERP) recordings in a visual attentional blink task wherein the T2-accompanying sound, when delivered, could be semantically neutral in addition to congruent or incongruent with respect to T2. Results: The behavioral data clearly showed that compared to the neutral sound, the congruent sound improved T2 discrimination during the blink to a higher degree while the incongruent sound improved it to a lesser degree. The T2-locked ERP data revealed that the early occipital cross-modal N195 component (192-228 ms after T2 onset) was uniquely larger in the congruent-sound condition than in the neutral-sound and incongruent-sound conditions, whereas the late parietal cross-modal N440 component (400-500 ms) was prominent only in the incongruent-sound condition. Discussion: These findings provide strong evidence that the modulating effect of audiovisual semantic congruency on the sound-induced alleviation of visual attentional blink contains not only a late incongruence-induced cost but also an early congruence-induced benefit, thereby demonstrating for the first time an unequivocal congruent-sound-induced benefit in alleviating the limitation of time-based visual attention.